DCS Comprehensive Health Plan

Inter-Rater Reliability Testing

Statement/Purpose

Inter-Rater Reliability (IRR) testing is utilized as a mechanism to monitor and evaluate comprehension of medical criteria, consistent decisions and to ensure accurate and consistent application of the criteria among all staff involved with the process.

Inter-Rater Reliability testing is conducted in an effort to:

• Minimize variation in the application of clinical guidelines;

• Evaluate staff ability to identify potentially avoidable utilization;

• Evaluate staff ability to identify quality of care concerns;

• Evaluate staff ability to triage, systematically examine and finalize quality of care concerns;

• Target specific areas most in need of improvement; and

• Identify staff needing additional training

Definitions

Inter-Rater Reliability (IRR) Testing: A process, based on systematic review, that determines congruence in outcomes among tasks. This process includes the evaluation of Evidence-Based criteria application, and navigation and knowledge of agency policies and procedures.

Policy

Inter-Rater Reliability (IRR) testing is completed annually by all DCS CHP staff and contracted MCO staff who participate in medical necessity determinations. Medical necessity determinations include, but are not limited to, conducting prior authorization, concurrent review, and retrospective review.wa

Procedure

Testing

IRR tests occurs for all staff involved in authorization decisions, at least annually.

Instructions are provided to staff for accessing and completing IRR testing, including time frames for completion.

IRR testing includes individual case scenarios, as well as cases developed to review highly specialized criteria such as behavioral health or transplant related areas.

Once testing is complete scores are tabulated to determine clinician performance.

Aggregate data is collected for presentation to the Chief Medical Officer and to the Quality Management/Performance Improvement (QM/PI) Committee.

Testing Frequency

IRR testing is conducted annually at a minimum, or more often to ensure process fidelity and consistency in decision making.

Testing for newly hired staff is conducted upon completion of training and annually thereafter.

Compliance

Clinical staff should achieve or exceed 90% consistency with determinations. Individual education plans are developed for those staff who do not meet the expected rate.

Results and Reporting

Individual IRR testing results are kept confidential.

Collective results are reported to DCS CHP and summarized for discussion at the Medical Management (MM) and/or Quality Management/Performance Improvement (QM/PI) committee meetings to identify opportunities for improvement and additional training needs when trends are identified.

DCS CHP’s contracted Managed Care Organization (MCO) reports results of IRR testing of the MCO clinical staff. Any DCS CHP staff that participate in the UM decision making are also included in the MCO IRR process.